Visual vehicle tracking is an important topic in intelligent transportation systems. Intersections are challenging locations for visual systems to track vehicles which are simultaneously moving in different directions. In addition, normal traffic flow may change at intersections due to accidents. Congestion, occlusion and undetermined motion flows are the nominated challenging issues of vehicle tracking at intersections. In this paper, a method for tracking multiple vehicles is proposed considering the vehicle motion directions to overcome undetermined motion flows. For this purpose, a Multilayer Model is presented, which assigns each motion flows to distinct layers. Moreover, we introduce different neighborhoods for various layers considering the regular motion flows in a layer. Hence, vehicles entering from the same side of intersection with the same motion direction are assigned to the same layer. Then the tracking is performed on different layers separately. In special cases such as vehicles crossing each other, misdetections or occlusion, the proposed tracking method can predict the vehicles tracks by using the stored tracking history and considering neighborhoods in that layer. Experimental results show consistency between proposed tracking method results and ground truth, also outperformance of other tracking methods in tracking vehicles crossing the intersection.